Imaging system, image processing method, and image processing program

ABSTRACT

A favorable noise reduction process that is optimized for capturing conditions and that prevents the occurrence of residual image components is enabled. Provided is an imaging system including: a first extraction section that extracts a local region that includes a pixel of interest from an image signal; a second extraction section that extracts, from another image signal captured at a different time, a local region located at almost the same position as said local region; a first noise reduction section that performs a noise reduction process by using the local regions; a noise estimation section that estimates an amount of noise included in the pixel of interest; a residual image detection section that detects a residual image component included in the local region based on the estimated amount of noise; and a second noise reduction section that performs a noise reduction process based on the detected residual image component.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to imaging systems, image processingmethods, and image processing programs that reduce noise components inimage signals captured in time series.

This application is based on Japanese Patent Application No.2008-167864, the content of which is incorporated herein by reference.

2. Description of Related Art

In general, noise components are included in digitized image signalscaptured by an image pickup device and by an analog circuit and ananalog-to-digital converter which are both associated therewith. A noisereduction process is performed to reduce the noise components includedin the image signals to obtain high-definition images. For example,various three-dimensional noise reduction processes have been proposedin which moving images are treated as temporally-continuous imageframes, and noise components that are not temporally correlated arereduced.

For example, in a three-dimensional noise reduction process, adifferential signal is obtained through subtraction between the currentimage signal and a past image signal, and a value obtained bymultiplying the differential signal between the frames by a constantfactor is added to or subtracted from the current image signal, therebyreducing noise. With this three-dimensional noise reduction process,although a noise-reduction effect can be obtained, image deteriorationsuch as residual images and tailing (hereinafter, referred to asresidual image components) may occur in a moving part of the imagebecause the differential signal includes not only noise components butalso motion components.

In this case, motion information obtained from the differential signaletc. is used to calculate the above-mentioned factor. The factor isreduced for the moving part to lower the noise reduction effect toreduce residual image components.

As an example of such noise reduction processes, in Japanese UnexaminedPatent Application, Publication No. HEI 6-62283, data temporally andspatially correlated to input pixel data is searched for to reducenoise. In Japanese Unexamined Patent Application, Publication No. HEI6-62283, a nonlinear filter is used that assigns a large weight to apixel having a value close to that of the input pixel data, assigns asmall weight to a pixel having a value far from that of the input pixeldata, and averages the pixel values. With this filter, a noise reductionprocess that corresponds to the movement of an object is performed.

As another example, in Japanese Unexamined Patent Application,Publication No. 2003-219208, the maximum value and the average value aredetected among differential data from a preceding frame, thedifferential data being obtained for a predetermined period of time;setting information used to generate correction data and a controlsignal used to control the noise reduction process on and off aregenerated from the detected maximum value and average value; the amountof noise is detected with precision; and noise reduction is performed.In Japanese Unexamined Patent Application, Publication No. 2002-33942,the difference in pixel signal between a plurality of frames is detectedas the degree of movement; the gain of cyclic noise is set based on theresult; and noise reduction is performed.

As still another example, in Japanese Unexamined Patent Application,Publication No. 2005-347821, an image signal level and a movementdetection result obtained from the frame difference are used to controla cyclic factor of noise, thereby performing a noise reduction processcorresponding to the brightness of an image.

As still another example, Japanese Unexamined Patent Application,Publication No. 9-81754 discloses an apparatus in which the degree ofmovement is compared from the difference between frames; comparisonresults are checked by a majority decision circuit; a variation of thecomparison results is corrected; the result is used as a movement signalto decide a cyclic factor of noise; and movement is detectedsuccessfully to perform noise reduction.

As described above, in the conventional three-dimensional noisereduction processes, the difference value from the current image signalis used to control the cyclic factor of noise to reduce residual imagecomponents detected when the image includes motion components.

However, the differential signal includes both the difference in motioncomponent between images and the difference in noise component includedin image signals. Therefore, when the sensitivity of detection of motioncomponents is increased, the sensitivity of detection of the differencein noise components is decreased. The difference in noise components isthus falsely detected as motion components, and thus a goodnoise-reduction effect cannot be obtained. On the other hand, when thesensitivity of detection of motion components is decreased, motioncomponents are falsely detected as noise components, leading to thedisadvantage that residual image components occur in the original movingpart.

Further, for example, when an image signal having a largenoise-component value, such as that obtained when the gain of an inputimage-signal level is increased, is input, the difference in noisecomponents is falsely detected as motion components, leading to aproblem in that a noise-reduction effect cannot be obtained and a goodnoise reduction cannot be performed.

BRIEF SUMMARY OF THE INVENTION

The present invention provides imaging systems, image processingmethods, and image processing programs allowing a superior noisereduction process that is optimized corresponding to capturingconditions and that prevents the occurrence of residual images.

According to a first aspect, the present invention provides an imagingsystem that applies noise reduction processing to image signals capturedin time series via an image pickup block, including: a first extractionsection that extracts a local region that includes a pixel of interestfrom an image signal to be processed; a second extraction section thatextracts, from another image signal captured at a different time fromthe image signal to be processed, a local region located at almost thesame position as the local region extracted by the first extractionsection; a first noise reduction section that applies a first noisereduction process to the local region extracted by the first extractionsection, by using the local region extracted by the second extractionsection; a noise estimation section that estimates an amount of noiseincluded in the pixel of interest based on the local region that hasbeen subjected to the first noise reduction process in the first noisereduction section; a residual image detection section that detects aresidual image component included in the local region that has beensubjected to the first noise reduction process and that is output fromthe first noise reduction section, based on the amount of noiseestimated by the noise estimation section; and a second noise reductionsection that performs a second noise reduction process on the pixel ofinterest based on the residual image component detected by the residualimage detection section.

According to a second aspect, the present invention provides an imageprocessing method of applying noise reduction processing to imagesignals captured in time series by an image pickup block, including: afirst step of extracting a local region that includes a pixel ofinterest from an image signal to be processed; a second step of storinga predetermined number of image signals; a third step of extracting,from another image signal captured at a different time from the imagesignal to be processed, a local region located at almost the sameposition as the local region extracted in the first step; a fourth stepof applying a first noise reduction process to the local regionextracted in the first step, by using the local region extracted in thethird step; a fifth step of estimating an amount of noise included inthe pixel of interest based on the local region that has been subjectedto the first noise reduction process in the fourth step; a sixth step ofdetecting a residual image component included in the local region thathas been subjected to the first noise reduction process in the fourthstep, based on the amount of noise estimated in the fifth step; and aseventh step of performing a second noise reduction process on the pixelof interest based on the residual image component detected in the sixthstep.

According to a third aspect, the present invention provides an imageprocessing program for causing a computer to apply noise reductionprocessing to image signals captured in time series by an image pickupblock, the image processing program causing the computer to execute: afirst process of extracting a local region that includes a pixel ofinterest from an image signal to be processed; a second process ofstoring a predetermined number of image signals; a third process ofextracting, from another image signal captured at a different time fromthe image signal to be processed, a local region located at almost thesame position as the local region extracted in the first process; afourth process of applying a first noise reduction process to the localregion extracted in the first process, by using the local regionextracted in the third process; a fifth process of estimating an amountof noise included in the pixel of interest based on the local regionthat has been subjected to the first noise reduction process in thefourth process; a sixth process of detecting a residual image componentincluded in the local region that has been subjected to the first noisereduction process in the fourth process, based on the amount of noiseestimated in the fifth process; and a seventh process of performing asecond noise reduction process on the pixel of interest based on theresidual image component detected in the sixth process.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a schematic configuration diagram showing the entireconfiguration of an imaging system according to a first embodiment ofthe present invention.

FIG. 2A is an explanatory diagram of an array of color-differentialline-sequential complementary filters used in the imaging system shownin FIG. 1.

FIG. 2B is an explanatory diagram related to local-region extractionfrom an even-field signal obtained with the color-differentialline-sequential complementary filters used in the imaging system shownin FIG. 1.

FIG. 2C is an explanatory diagram related to local-region extractionfrom an odd-field signal obtained with the color-differentialline-sequential complementary filters used in the imaging system shownin FIG. 1.

FIG. 3 is a block configuration diagram showing a noise reductionsection of the imaging system shown in FIG. 1.

FIG. 4 is a graph showing the relationship between estimated amounts ofnoise and a signal level in the imaging system shown in FIG. 1.

FIG. 5 is a block configuration diagram showing a residual imagedetection section of the imaging system shown in FIG. 1.

FIG. 6 is a block configuration diagram showing a second noise reductionsection of the imaging system shown in FIG. 1.

FIG. 7 is a flowchart showing the flow of the entire processingperformed in the imaging system shown in FIG. 1.

FIG. 8 is a flowchart showing the flow of a noise reduction processperformed in the imaging system shown in FIG. 1.

FIG. 9 is a schematic configuration diagram showing the entireconfiguration of an imaging system according to a second embodiment ofthe present invention.

FIG. 10 is a block configuration diagram showing a noise reductionsection of the imaging system shown in FIG. 9.

FIG. 11 is an explanatory diagram of the relationship between each ofthe arrays of RGB filters and local-region extraction, in the imagingsystem shown in FIG. 9.

FIG. 12 is a block configuration diagram showing a residual imagedetection section of the imaging system shown in FIG. 9.

FIG. 13 is a block configuration diagram showing a second noisereduction section of the imaging system shown in FIG. 9.

FIG. 14A is a graph showing the relationship between a weighting factorand a calculated value output from the residual image detection sectionin the imaging system shown in FIG. 9.

FIG. 14B is a graph showing the relationship between a weighting factorand a calculated value output from the residual image detection sectionin the imaging system shown in FIG. 9.

FIG. 14C is a graph showing the relationship between a weighting factorand a calculated value output from the residual image detection sectionin the imaging system shown in FIG. 9.

DETAILED DESCRIPTION OF THE INVENTION First Embodiment

An imaging system according to a first embodiment of the presentinvention will be described below with reference to FIGS. 1 to 8.

FIG. 1 is a schematic configuration diagram showing the entireconfiguration of the imaging system according to this embodiment. FIG.2A is an explanatory diagram of a color-differential line-sequentialcomplementary filter array. FIG. 2B is an explanatory diagram related tolocal-region extraction from an even-field signal. FIG. 2C is anexplanatory diagram related to local-region extraction from an odd-fieldsignal. FIG. 3 is a block configuration diagram showing a noisereduction section of the imaging system shown in FIG. 1. FIG. 4 showsthe relationship between the amounts of noise and a signal level. FIG. 5is a block configuration diagram showing a residual image detectionsection of the imaging system shown in FIG. 1. FIG. 6 is a blockconfiguration diagram showing a second noise reduction section of theimaging system shown in FIG. 1. FIG. 7 is a flowchart showing the flowof the entire processing performed in the imaging system shown inFIG. 1. FIG. 8 is a flowchart showing the flow of a noise reductionprocess performed in the imaging system shown in FIG. 1.

As shown in FIG. 1, the imaging system according to this embodimentincludes an image pickup device (image pickup block) 101, ananalog-to-digital converter 102, a Y/C separation section 103, a noisereduction section 104, a signal processing section 105, an outputsection 106, a control section 107, and an external interface section108.

The image pickup device 101 applies photoelectric conversion to a formedoptical image and outputs it as an analog image signal. Note that, inthis embodiment, it is assumed that a single-plane image pickup devicein which a color-differential line-sequential complementary filter suchas that shown in FIG. 2A is disposed in front is used as the imagepickup device 101. Note that the image pickup device 101 is not limitedto this, and a single-plane image pickup device in which a Bayer primarycolor filter is disposed in front, or the like, may be used as the imagepickup device 101.

FIG. 2A shows the configuration of the color-differentialline-sequential complementary filters. In this color-differentialline-sequential complementary filters, 2×2 pixels are treated as a baseunit, and Cyan (Cy), Magenta (Mg), Yellow (Ye), and Green (G) arearranged at respective pixels. However, the positions of Mg and G areinverted every line.

An image signal transferred from the image pickup device 101 is composedof two field signals separated into even lines and odd lines obtained byadding two vertically adjacent lines in the manner shown in FIG. 2A. Aneven-field signal and an odd-field signal are combined to obtain oneimage signal. Hereinafter, a plurality of image signals that aresequentially output are referred to as a video signal, and one imagesignal is just referred to as an image signal.

Under the control of the control section 107, the analog-to-digitalconverter 102 converts an analog image signal output from the imagepickup device 101 into a digital image signal. A video signal capturedthrough the image pickup device 101 is sequentially output as an analogsignal at predetermined time intervals. The analog signal is convertedinto a digital signal in the analog-to-digital converter 102, and thedigital signal is transferred to the Y/C separation section 103.

Under the control of the control section 107, the Y/C separation section103 calculates a luminance signal Y and color-difference signals Cb andCr from the even-field signal and the odd-field signal, as shown inExpression 1.

Y=G+Mg+Ye+Cy

Cb=(Mg+Cy)−(G+Ye)

Cr=(Mg+Ye)−(G+Cy)   [Expression 1]

Under the control of the control section 107, the noise reductionsection 104 performs a noise reduction process by using the image signaltransferred from the Y/C separation section 103. The image signal thathas been subjected to the noise reduction process is transferred to thesignal processing section 105.

As shown in FIG. 3, the noise reduction section 104 includes, forexample, a first region extraction section (first extraction section)201, a second region extraction section (second extraction section) 202,a first noise reduction section 203, a noise estimation section 204, aresidual image detection section 205, a second noise reduction section206, and a frame memory (storage section) 207.

FIG. 3 is a block configuration diagram showing the noise reductionsection of the imaging system shown in FIG. 1.

Further, the Y/C separation section 103 is connected to the signalprocessing section 105 via the first region extraction section 201, thefirst noise reduction section 203, and the second noise reductionsection 206. The first region extraction section 201 is also connectedto the residual image detection section 205 and the second noisereduction section 206.

The first noise reduction section 203 is also connected to the noiseestimation section 204 and the residual image detection section 205. Thenoise estimation section 204 is connected to the residual imagedetection section 205. The residual image detection section 205 isconnected to the second noise reduction section 206. The second noisereduction section 206 is also connected to the frame memory 207. Theframe memory 207 is connected to the first noise reduction section 203via the second region extraction section 202.

Under the control of the control section 107, the signal processingsection 105 reads the image signal that has been subjected to the noisereduction process from the noise reduction section 104, applies knownprocesses, such as an interpolation process, an enhancement process, anda compression process, to the read image signal, and transfers the imagesignal that has been subjected to those processes to the output section106.

The output section 106 records and stores the video signal output fromthe signal processing section 105, in a recording medium such as amemory card.

The control section 107 is implemented by a microcomputer, for example,and is bi-directionally connected to the analog-to-digital converter102, the Y/C separation section 103, the noise reduction section 104,the signal processing section 105, the output section 106, and theexternal interface section 108, so as to control the whole of an imageprocessing apparatus that includes those sections.

Further, the control section 107 is bi-directionally connected to thefirst region extraction section 201, the second region extractionsection 202, the first noise reduction section 203, the noise estimationsection 204, the residual image detection section 205, and the secondnoise reduction section 206, so as to control those sections.

The external interface section 108 is an interface for receiving aninput from a user to this image processing apparatus to which an imagingsystem is applied. The external interface section 108 includes a powerswitch used for turning the power on or off, a shutter button used tostart an image pickup operation, and a mode switch button used to switchthe mode among an image pickup mode and other various modes. The usercan specify the gain etc. through the external interface section 108.Then, the external interface section 108 outputs the receivedinformation to the control section 107.

From an image signal to be subjected to the noise reduction process, thefirst region extraction section 201 sequentially extracts a local regionthat includes a pixel of interest and adjacent pixels located adjacentto the pixel of interest, under the control of the control section 107,and outputs it to the first noise reduction section 203, the residualimage detection section 205, and the second noise reduction section 206.

In this embodiment, it is assumed that the local region is formed of 5×5pixels, as shown in FIGS. 2B and 2C. The pixel region is not limited tothis, and any pixel region may be specified as a local region. Forexample, a square region formed of 3×3 pixels or a rectangular regionformed of 7×5 pixels may be specified as a local region.

As another example, a square region formed of 4×4 pixels or arectangular region formed of 4×6 pixels may be specified as a localregion. In this case, since the center portion of the square region isformed of not one pixel but a plurality of pixels, one of the pixelsserves as the pixel of interest.

The size of a local region may be specified in advance according to auser instruction. The luminance signal Y is included in all of the 5×5pixels, as shown in FIG. 2B for the even field and as shown in FIG. 2Cfor the odd field. In contrast, color-difference signals Cr and Cb arenot included in all of the 5×5 pixels. Specifically, thecolor-difference signal Cr is included in 5×3 pixels in the even field,as shown in FIG. 2B, and in 5×2 pixels in the odd field, as shown inFIG. 2C. Further, the color-difference signal Cb is included in 5×2pixels in the even field, as shown in FIG. 2B, and in 5×3 pixels in theodd field, as shown in FIG. 2C.

In such local-region configurations, the pixel of interest (for example,the pixel located at the center of a local region) to be subjected tothe noise reduction process includes either the luminance signal Y andthe color-difference signal Cr (the color-difference signal Cb is notsubjected to the process), or the luminance signal Y and thecolor-difference signal Cb (the color-difference signal Cr is notsubjected to the process). In the examples shown in FIGS. 2B and 2C, theformer case corresponds to the even field, and the latter casecorresponds to the odd field. In contrast, however, if the pixel ofinterest is located at a different position, the latter case maycorrespond to the even field and the former case may correspond to theodd field.

From a past image signal that has been subjected to the noise reductionprocess and that is stored in the frame memory 207, the second regionextraction section 202 sequentially extracts a local region thatincludes a pixel located in the same spatial position as theabove-mentioned pixel of interest, and adjacent pixels located adjacentto that pixel, under the control of the control section 107, and outputsit to the first noise reduction section 203. Note that only the pixellocated at the same spatial position as the pixel of interest may beextracted, or only pixels located adjacent to the same spatial positionas the pixel of interest may be extracted.

In the above-described configuration, at a point in time when the noisereduction section 104 applies the noise reduction process to an imagesignal obtained at time T, the first region extraction section 201extracts a local region from the image signal obtained at time T, andthe second region extraction section 202 extracts a local region from animage signal that was obtained before time T.

Note that the image signal obtained at time T may be stored andsubjected to the noise reduction process based on another image signalobtained at a different time after time T.

Under the control of the control section 107, the first noise reductionsection 203 performs a three-dimensional noise reduction process basedon the local region obtained at time T that is transferred from thefirst region extraction section 201 and based on the local regionobtained before time T that has been subjected to the noise reductionprocess and that is transferred from the second region extractionsection 202. In this embodiment, it is assumed that a nonlinear filteris used in which, with respect to the pixel of interest, the adjacentpixels obtained at time T, and adjacent pixels (including the pixellocated at the same spatial position as the pixel of interest) obtainedbefore time T, weighted addition is applied with a large weight beingassigned to an adjacent pixel having a value close to that of the pixelof interest and with a small weight being assigned to an adjacent pixelhaving a value far from that of the pixel of interest. The first noisereduction section 203 outputs to the noise estimation section 204 aluminance signal NRy and color-difference signals NRcb and NRcr of thepixel of interest that are obtained after the three-dimensional noisereduction process.

Under the control of the control section 107, the noise estimationsection 204 estimates the amounts of noise Ns (s=y, cb, cr) in the pixelof interest by using the signal values NRs (s=y, cb, cr) of the pixel ofinterest that are obtained after the three-dimensional noise reductionprocess and that are transferred from the first noise reduction section203.

Referring to FIG. 4, estimation of the amounts of noise will bedescribed. FIG. 4 shows the relationship between the amounts of noise Ns(s=y, cb, cr) and a signal level L. As shown in FIG. 4, the amounts ofnoise Ns increase in quadratically with respect to the signal level L.When the signal level L is set to the signal values NRs (s=y, cb, cr)obtained after the three-dimensional noise reduction process,constituting such a specific noise model by using a quadratic functionyields Expression 2.

Ny=αy×NRy ² +βy×NRy+γy

Ncb=αcb×NRcb ² +βcb×NRcb+γcb

Ncr=αcr×NRcr ² +βcr×NRcr+γcr   [Expression 2]

where αs, βs, and γs are constant terms. The noise estimation section204 uses models such as those shown in Expression 2 to estimate theamounts of noise Ns from the signal values NRs (s=y, cb, cr) of thepixel of interest that are obtained after the three-dimensional noisereduction process. A luminance noise amount Ny and color-differencenoise amounts Ncb and Ncr are output to the residual image detectionsection 205.

Under the control of the control section 107, the residual imagedetection section 205 detects residual image components included in thesignal values NRs (s=y, cb, cr) obtained after the three-dimensionalnoise reduction process, based on the estimated amounts of noise Ns(s=y, cb, cr). Judgment results Ts (s=y, cb, cr) indicating whetherresidual image components are included in the pixel of interest areoutput to the second noise reduction section 206. The judgment resultsTs indicate either that “residual image components are included” or that“residual image components are not included”.

Under the control of the control section 107, the second noise reductionsection 206 applies a two-dimensional noise reduction process to theluminance signal Y and the color-difference signals Cb and Cr of thepixel of interest by using the signal values NRs obtained after thethree-dimensional noise reduction process, and the judgment results Tsreceived from the residual image detection section 205. The second noisereduction section 206 outputs a luminance signal NR2 y andcolor-difference signals NR2 cb and NR2 cr of the pixel of interest thatare obtained after the two-dimensional noise reduction process, to thesignal processing section 105 and the frame memory 207.

Thus, the image signal that has been subjected to the noise reductionprocess is recorded in the frame memory 207 and will be used forprocessing for the next image signal.

Next, the detailed configuration of the residual image detection section205 will be described with reference to FIG. 5.

Under the control of the control section 107, a calculation section 301calculates absolute difference signal values Ss (s=y, cb, cr) betweenthe luminance signal value Y and the color-difference signal values Cband Cr of the pixel of interest in the local region, transferred fromthe first region extraction section 201, and the luminance signal valueNRy and the color-difference signal values NRcb and NRcr, obtained afterthe three-dimensional noise reduction process, and outputs the absolutedifference signal values Ss to a judgment section 303. The absolutedifference signal values Ss include noise components in the signalvalues of the pixel of interest or residual image components in thesignal values obtained after the three-dimensional noise reductionprocess.

Under the control of the control section 107, an adjustment section 302adjusts the estimated amounts of noise Ns output from the noiseestimation section 204 and outputs adjusted estimated amounts of noiseCNs (s=y, cb, cr) to the judgment section 303. The adjustment section302 reads the judgment results of a pixel located adjacent to the pixelof interest from a judgment-result storage section 304. It is assumedthat one adjacent pixel is used, but the number of adjacent pixels isnot limited to this. As another example, four adjacent pixels or eightadjacent pixels may be used. When the judgment results Ts of theadjacent pixel indicate that “residual image components are included”,the adjustment section 302 adjusts the estimated amounts of noise Ns tomake them smaller.

Under the control of the control section 107, the judgment section 303compares the absolute difference signal values Ss output from thecalculation section 301 with the adjusted estimated amounts of noise CNsoutput from the adjustment section 302, to judge whether residual imagecomponents are included in the noise-reduced luminance signal values NRyand in the noise-reduced color-difference signal values NRcb and NRcr.

When the absolute difference signal values Ss are larger than theadjusted estimated amounts of noise CNs, it is judged that residualimage components are included in the signal values NRs, obtained afterthe three-dimensional noise reduction process. Then, the judgmentresults Ts, indicating whether residual image components are included inthe pixel of interest, are set to indicate that “residual imagecomponents are included”. On the other hand, when the absolutedifference signal values Ss are smaller than the adjusted estimatedamounts of noise CNs, it is judged that residual image components arenot included in the signal values NRs, obtained after thethree-dimensional noise reduction process. Then, the judgment results Tsare set to indicate that “residual image components are not included”.The judgment results Ts are output to the second noise reduction section206 and the judgment-result storage section 304.

Next, the detailed configuration of the second noise reduction section206 will be described with reference to FIG. 6.

Under the control of the control section 107, a switching section 401switches the luminance signal Y and the color-difference signals Cb andCr of the pixel of interest, transferred from the first regionextraction section 201, to the signal values NRs of the pixel ofinterest, obtained after the three-dimensional noise reduction processand transferred from the first noise reduction section 203.

Specifically, when the judgment results Ts obtained by the residualimage detection section 205 indicate that “residual image components arenot included”, the switching section 401 substitutes the signal valuesNRs of the pixel of interest, obtained after the three-dimensional noisereduction process, into the luminance signal value Y and thecolor-difference signal values Cb and Cr of the pixel of interest, asshown in Expression 3.

Y=NRy

Cb=NRcb

Cr=NRcr   [Expression 3]

When the judgment results Ts obtained by the residual image detectionsection 205 indicate that “residual image components are included”, theswitching section 401 does not perform the substitution process for theluminance signal value Y and the color-difference signal values Cb andCr of the pixel of interest. Then, the switching section 401 outputs theluminance signal value Y and the color-difference signal values Cb andCr of the pixel of interest to a two-dimensional noise reduction section402.

Under the control of the control section 107, the two-dimensional noisereduction section 402 performs a two-dimensional noise reduction processbased on the pixel values of the pixel of interest that are transferredfrom the switching section 401, the pixel values of a pixel locatedadjacent to the pixel of interest that are stored in a line memory (notshown) included in the two-dimensional noise reduction section 402, andthe judgment results Ts transferred from the residual image detectionsection 205. In this embodiment, it is assumed that a nonlinear filteris used in which, with respect to the pixel of interest and adjacentpixels, weighted addition is performed with a large weight beingassigned to an adjacent pixel that is spatially close to the pixel ofinterest and with a small weight being assigned to an adjacent pixelthat is spatially far from the pixel of interest; and further, weightedaddition is performed with a large weight being assigned to an adjacentpixel having a value close to that of the pixel of interest and with asmall weight being assigned to an adjacent pixel having a value far fromthat of the pixel of interest.

When the judgment results Ts of the pixel of interest indicate that“residual image components are included”, weighted addition is performedwith a large weight being assigned to an adjacent pixel. Thetwo-dimensional noise reduction section 402 outputs a luminance signalNR2 y and color-difference signals NR2 cb and NR2 cr of the pixel ofinterest that are obtained after the two-dimensional noise reductionprocess to the signal processing section 105 and the frame memory 207.

Note that a description has been given of an image pickup apparatus inwhich image pickup parts, such as the image pickup device 101 and theanalog-to-digital converter 102, are integrated, as an example of avideo signal processing apparatus; however, the video signal processingapparatus is not limited to this configuration. For example, aconfiguration may be used in which a video signal captured by aseparately provided image pickup part is recorded in a recording medium,such as a memory card, as unprocessed raw data; accompanyinginformation, such as image pickup conditions and data obtained at thetime of image pickup, is also recorded in the recording medium as headerinformation; and the information recorded in the recording medium isread and processed by the video signal processing apparatus.

When the video signal processing apparatus has a separately providedimage pickup part, transmission of information to the video signalprocessing apparatus is not limited to that performed through arecording medium; it can, of course, be performed through wired orwireless communication lines.

Further, in the above description, it is assumed that the processing isperformed by hardware; however, the way the processing is performed isnot limited to this configuration. For example, a configuration may beused in which a signal transferred from the image pickup device 101 isoutput as unprocessed raw data together with header information, such asa configuration selected during capturing, the gain, and the amount of achange in luminance level, transferred from the control section 107; andthe signal is processed in a computer by a video signal processingprogram serving as separate software.

Referring to FIG. 7, a description will be given of the flow ofprocessing performed when the computer executes the video signalprocessing program.

When the processing is started, a video signal and header information,such as gain, are read (Step 1).

Next, each image signal included in the read video signal is separatedinto a luminance signal and color-difference signals, as shown inExpression 1 (Step 2).

As will be described later with reference to FIG. 8, the noise reductionprocess is applied to the luminance signal and the color-differencesignals (Step 3).

Further, the image signal that has been subjected to the noise reductionprocess is output (Step 4).

Known signal processes, such as a gradation conversion process, anenhancement process, and a compression process, are applied to the imagesignal that has been subjected to the noise reduction process (Step 5).

Next, the image signal that has been subjected to the signal processesis output (Step 6).

It is judged whether all image signals included in the video signal havebeen processed (Step 7). If it is judged that all image signals includedin the video signal have not been processed, the flow returns to Step 2,and the above-described processing is repeatedly performed for the nextimage signal.

On the other hand, if it is judged that all image signals included inthe video signal have been processed, the processing ends.

Next, the noise reduction process performed in Step 3 of FIG. 7 will bedescribed in detail with reference to FIG. 8.

When the process is started, a local region that includes a pixel ofinterest is extracted from the image signal that is being subjected tothe noise reduction process, as shown in FIGS. 2B and 2C (Step 11).

Then, a past image signal that was subjected to the noise reductionprocess is input (Step 12).

Further, a local region that includes a pixel located at the samespatial position as the pixel of interest is extracted from the pastimage signal (Step 13).

Next, the three-dimensional noise reduction process is performed basedon the local region that includes the pixel of interest and based on thepast local region (Step 14).

The amounts of noise are estimated, as shown in Expression 2, based onthe signal values of the pixel of interest that are obtained after thethree-dimensional noise reduction process (Step 15).

The judgment results of an adjacent pixel located adjacent to the pixelof interest are read (Step 16).

The amounts of noise are corrected based on the judgment results of theadjacent pixel (Step 17).

Next, the absolute difference values between the signal values of thepixel of interest that are obtained through the separation in Step 2 andthe signal values of the pixel of interest that are obtained after thethree-dimensional noise reduction process that are calculated in Step 14are calculated (Step 18).

It is judged whether the absolute difference values for the pixel ofinterest are smaller than the corrected estimated amounts of noise (Step19).

When it is judged in Step 19 that the absolute difference values for thepixel of interest are smaller than the corrected estimated amounts ofnoise, the signal values of the pixel of interest that are obtainedafter the three-dimensional noise reduction process are substituted intothe signal values of the pixel of interest (Step 20).

Next, the two-dimensional noise reduction process is performed based onthe local region that includes the pixel of interest (Step 21).

Then, it is judged whether all local regions that can be extracted fromthe image signal have been processed (Step 22). If it is judged that alllocal regions that can be extracted from the image signal have not beenprocessed, the flow returns to Step 11, and the next local region isextracted and processed as described above.

On the other hand, if it is judged in Step 22 that all local regionsthat can be extracted from the image signal have been processed, theflow returns to the processing shown in FIG. 7.

Note that, although the number of past image signals used for the noisereduction process is not limited in the above description, only an imagesignal immediately preceding the image signal that is being subjected tothe process may be used or a plurality of image signals captured beforethe image signal that is being subjected to the process may be used. Inthe latter case, however, the frame memory 207 requires a storagecapacity capable of storing the signals of a plurality of frames, thesecond region extraction section 202 performs the corresponding processfor the number of frames that are to be subjected to the process, andthe first noise reduction section 203 performs the corresponding processfor the plurality of frames.

Although it is assumed that the factors, αs, βs, and γs (s=y, cb, cr),in Expression 2 are just constant terms in the above description, if thefactors are constant terms that depend on the temperature, the gain,etc. of the image pickup device 101 obtained when the image signal wascaptured, it is also possible to deal with such factors.

According to this embodiment, since the amounts of noise are estimatedwith high precision corresponding to factors that dynamically change atthe time of capturing, such as the movement of an object, the signallevel, and the luminance level, it is possible to apply the optimumnoise reduction to the entire screen. As a result, a high-definitionvideo signal can be obtained.

At this time, since the amounts of noise included in an image signal tobe processed are estimated based on a past image signal from which noisecomponents have been eliminated, the precision in estimating the amountsof noise can be improved.

Further, since the amount of luminance noise and the amounts ofcolor-difference noise are estimated independently, the precision inestimating them can be improved.

Since the noise reduction process is performed with a noise range beingspecified based on the amounts of noise, signal values that are totallydifferent from those of the original signal are not obtained, which isadvantageous in that it leads to good capability of maintaining theoriginal signal. When it is judged that the amounts of noise of thepixel of interest fall within the noise range, the pixel values obtainedafter the three-dimensional noise reduction process are used as those ofthe pixel of interest. When it is judged that the amounts of noise ofthe pixel of interest do not fall within the noise range, the values ofthe pixel of interest are corrected based on the amounts of noise.Therefore, it is possible to prevent the occurrence of a discontinuitycaused by the noise reduction process, thus obtaining a high-definitionsignal.

The signal obtained after the noise reduction process is output as anactual signal, and then, various signal processes are applied thereto.Thus, compatibility with conventional processing systems is maintained,leading to an advantage in that a combination with various systems isallowed.

Since a luminance signal and color-difference signals are obtainedcorresponding to the color-differential line-sequential complementaryfilter array, high-speed processing is enabled.

Further, the estimated amounts of noise Ns are corrected based on theresidual image judgment results of an adjacent pixel. Specifically, whenresidual image components that are spatially correlated are included inthe adjacent pixel, the noise range is narrowed, in other words, therange to which the three-dimensional noise reduction process (seeExpression 3) is applied is narrowed. Therefore, the residual imagecomponents can be eliminated with high precision from the image that hasbeen subjected to the three-dimensional noise reduction process. On theother hand, when the residual image components are not included in theadjacent pixel, the noise range is not changed, and thethree-dimensional noise reduction process can be effectively applied toa static object. Thus, a high-definition signal can be obtained.

When it is judged that residual image components are included in theimage signal of the pixel of interest that has been subjected to thethree-dimensional noise reduction process, adjustment is made such thata weight for the two-dimensional noise reduction process is made larger.Therefore, it is possible to prevent the occurrence of a discontinuitycaused by the noise reduction process, thus obtaining a high-definitionsignal.

According to this embodiment, based on an image signal that was obtainedat a different time and stored in the storage section, the amounts ofnoise of a luminance signal and color-difference signals are modeledcorresponding to factors that dynamically change at the time ofcapturing, such as the signal level and gain, and the amounts of noiseare estimated based on the models. Then, based on the estimated amountsof noise, residual image components included in the local region thathas been subjected to the three-dimensional noise reduction process inthe first noise reduction section are detected, and the two-dimensionalnoise reduction process is applied to reduce the residual imagecomponents. As a result, it is possible to perform a noise reductionprocess that suppresses residual image components to obtain ahigh-definition image signal.

Note that the present invention is not limited to the above-describedembodiment, and the components can be modified and embodied in the phaseof reduction to practice, without departing from the scope thereof.Further, a plurality of components disclosed in the above-describedembodiment can be appropriately combined to form various aspects of theinvention. For example, some components may be deleted from all of thecomponents shown in the embodiment. Furthermore, components fromdifferent embodiments can be appropriately combined. Variousmodifications and applications are possible without departing from thegist of the invention.

Second Embodiment

An imaging system according to a second embodiment of the presentinvention will be described below with reference to FIG. 9 to FIGS. 14A,14B, and 14C.

In the second embodiment, a description will be given mainly of thedifferences from the first embodiment, and identical reference symbolsare given to similar parts and a description thereof will be omitted.

FIG. 9 is a schematic configuration diagram showing the entireconfiguration of an imaging system according to the second embodiment.FIG. 10 is a block configuration diagram showing a noise reductionsection of the imaging system shown in FIG. 9. FIG. 11 is an explanatorydiagram of the relationship between each of the arrays of RGB filtersand local-region extraction. FIG. 12 is a block configuration diagramshowing a residual image detection section of the imaging system shownin FIG. 9. FIG. 13 is a block configuration diagram showing a secondnoise reduction section of the imaging system shown in FIG. 9. FIGS.14A, 14B, and 14C are graphs showing the relationship between aweighting factor and a calculated value output from the residual imagedetection section.

First, the flow of a signal in the imaging system shown in FIG. 9according to this embodiment will be described.

The imaging system shown in FIG. 9 is obtained by partially modifyingthe imaging system shown in FIG. 1 according to the first embodiment.

Specifically, in the imaging system, the image pickup device 101, shownin FIG. 1, is replaced with an R image pickup device 501, a G imagepickup device 502, and a B image pickup device 503; the noise reductionsection 104 is replaced with a noise reduction section 504; and the Y/Cseparation section 103 is removed. The R image pickup device 501, the Gimage pickup device 502, and the B image pickup device 503 are providedwith a dichroic prism for separating incident light flux into threecolor components of RGB. The other basic configurations are the same asthose of the first embodiment, and identical names and reference symbolsare assigned to the same configurations.

A description will be given below mainly of the differences. Thedichroic prism separates light flux into three colors, RGB, and guideslight having R components to the R image pickup device 501, light havingG components to the G image pickup device 502, and light having Bcomponents to the B image pickup device 503. Thus, the R image pickupdevice 501, the G image pickup device 502, and the B image pickup device503 each subject the light to photoelectric conversion and output it asan analog image signal. The R image pickup device 501, the G imagepickup device 502, and the B image pickup device 503 are connected tothe analog-to-digital converter 102.

Next, an example configuration of the noise reduction section 504 willbe described with reference to FIG. 10.

The noise reduction section 504 shown in FIG. 10 is obtained bypartially modifying the noise reduction section 104 shown in FIG. 3according to the first embodiment. Specifically, in the noise reductionsection 504, the Y/C separation section 103 shown in FIG. 3 is replacedwith the analog-to-digital converter 102; the first region extractionsection 201 shown in FIG. 3 is replaced with a first region extractionsection 601; the second region extraction section 202 shown in FIG. 3 isreplaced with a second region extraction section 602; the c detectionsection 205 shown in FIG. 3 is replaced with a residual image detectionsection 605; and the second noise reduction section 206 shown in FIG. 3is replaced with a second noise reduction section 606. The other basicconfigurations are the same as those of the first embodiment, andidentical names and reference symbols are assigned to the sameconfigurations.

The analog-to-digital converter 102 is connected to the signalprocessing section 105 via the first region extraction section 601, thefirst noise reduction section 203, and the second noise reductionsection 606. The first region extraction section 601 is also connectedto the residual image detection section 605 and the second noisereduction section 606. The first noise reduction section 203 is alsoconnected to the noise estimation section 204 and the residual imagedetection section 605.

The noise estimation section 204 is connected to the residual imagedetection section 605. The residual image detection section 605 isconnected to the second noise reduction section 606. The second noisereduction section 606 is also connected to the frame memory 207. Theframe memory 207 is connected to the first noise reduction section 203via the second region extraction section 602.

Further, the control section 107 is bi-directionally connected to thefirst region extraction section 601, the second region extractionsection 602, the first noise reduction section 203, the noise estimationsection 204, the residual image detection section 605, and the secondnoise reduction section 606, and controls those sections.

From an image signal to be subjected to the noise reduction process, thefirst region extraction section 601 sequentially extracts a local regionthat includes a pixel of interest and adjacent pixels located adjacentto the pixel of interest, under the control of the control section 107,and outputs it to the first noise reduction section 203, the residualimage detection section 605, and the second noise reduction section 606.

In this embodiment, it is assumed that the local region is formed of 5×5pixels, as shown in FIG. 11. The pixel region is not limited to this,and any pixel region may be specified as a local region. For example, asquare region formed of 3×3 pixels or a rectangular region formed of 7×5pixels may be specified as a local region.

As another example, a square region formed of 4×4 pixels or arectangular region formed of 4×6 pixels may be specified as a localregion. In this case, since the center portion of the square region isformed of not one pixel but a plurality of pixels, one of the pixelsserves as the pixel of interest. The size of a local region may bespecified in advance according to a user instruction.

From a past image signal that has been subjected to the noise reductionprocess and that is stored in the frame memory 207, the second regionextraction section 602 sequentially extracts a local region thatincludes a pixel located in the same spatial position as the pixel ofinterest shown in FIG. 11, and adjacent pixels located adjacent to thatpixel, under the control of the control section 107, and outputs it tothe first noise reduction section 203.

Under the control of the control section 107, the second noise reductionsection 606 performs a two-dimensional noise reduction process by usingRGB signal values of the pixel of interest output from the first regionextraction section 601, noise-reduced RGB signal values of the pixel ofinterest output from the first noise reduction section 203, andcalculated values output from the residual image detection section 605.The second noise reduction section 606 outputs RGB signal values of thepixel of interest that are obtained after the two-dimensional noisereduction process to the signal processing section 105 and the framememory 207.

Next, the detailed configuration of the residual image detection section605 will be described with reference to FIG. 12.

The residual image detection section 605 shown in FIG. 12 is obtained bypartially modifying the residual image detection section 205 shown inFIG. 5 according to the first embodiment. Specifically, in the residualimage detection section 605, the judgment section 303 shown in FIG. 5 isreplaced with a judgment section 703. The other basic configurations arethe same as those of the first embodiment, and identical names andreference symbols are assigned to the same configurations.

Under the control of the control section 107, the judgment section 703compares the absolute difference signal values output from thecalculation section 301 with the adjusted estimated amounts of noiseoutput from the adjustment section 302, to judge whether residual imagecomponents are included in the noise-reduced RGB signals. The differencevalues between the absolute difference signal values and the adjustedestimated amounts of noise are calculated. When the absolute differencesignal values are larger than the adjusted estimated amounts of noise,the calculated values become positive. When the absolute differencesignal values are smaller than the adjusted estimated amounts of noise,the calculated values become negative.

When the calculated values are positive, it is judged that residualimage components are included in the signal subjected to thethree-dimensional noise reduction process, and judgment results,indicating whether residual image components are included in the pixelof interest, are set to indicate that “residual image components areincluded”. On the other hand, when the calculated values are negative,it is judged that residual image components are not included in thesignal subjected to the three-dimensional noise reduction process, andjudgment results are set to indicate that “residual image components arenot included”. The calculated values are output to the second noisereduction section 606. The judgment results are output to thejudgment-result storage section 304.

Next, the detailed configuration of the second noise reduction section606 will be described with reference to FIG. 13.

Under the control of the control section 107, a two-dimensional noisereduction section 801 performs the two-dimensional noise reductionprocess based on the local region that includes the pixel of interest,transferred from the first region extraction section 601.

In this embodiment, it is assumed that a nonlinear filter is used inwhich, with respect to the pixel of interest and adjacent pixels,weighted addition is performed with a large weight being assigned to anadjacent pixel that is spatially close to the pixel of interest and witha small weight being assigned to an adjacent pixel that is spatially farfrom the pixel of interest; and further, weighted addition is performedwith a large weight being assigned to an adjacent pixel having a valueclose to that of the pixel of interest and with a small weight beingassigned to an adjacent pixel having a value far from that of the pixelof interest. The two-dimensional noise reduction section 801 outputs RGBsignals NR2 s (s=r, g, b) of the pixel of interest that are obtainedafter the two-dimensional noise reduction process, to a combiningsection 803.

Under the control of the control section 107, a factor calculationsection 802 specifies weighting factors Ks (s=r, g, b) for the RGBsignals based on the calculated values transferred from the residualimage detection section 605.

When the calculated values for the pixel of interest are positive, theweighting factors Ks are made larger. On the other hand, when thecalculated values for the pixel of interest are negative, the weightingfactors Ks are made smaller. FIGS. 14A, 14B, and 14C show examples ofthe relationship between the calculated value and the weighting factorKs.

Under the control of the control section 107, the combining section 803combines the RGB signal values of the pixel of interest that aretransferred from the two-dimensional noise reduction section 801 and theRGB signal values NRs (s=r, g, b) of the pixel of interest that aretransferred from the first noise reduction section 203, based on theweighting factors Ks (s=r, g, b) output from the factor calculationsection 802. Output signal values NR3 s (s=r, g, b) are given byExpression 4.

NR3r=Kr×NR2r+(1−Kr)×NRr

NR3g=Kg×NR2g+(1−Kg)×NRg

NR3b=Kb×NR2b+(1−Kb)×NRb   [Expression 4]

Note that a description has been given of an image pickup apparatus inwhich image pickup parts, such as the R image pickup device 501, the Gimage pickup device 502, the B image pickup device 503, and theanalog-to-digital converter 102, are integrated, as an example of avideo signal processing apparatus; however, the video signal processingapparatus is not limited to this configuration. For example, aconfiguration may be used in which a video signal captured by aseparately provided image pickup part is recorded in a recording medium,such as a memory card, as unprocessed raw data; accompanyinginformation, such as image pickup conditions and data obtained at thetime of image pickup, is also recorded in the recording medium as headerinformation; and the information recorded in the recording medium isread and processed by the video signal processing apparatus.

When the video signal processing apparatus has a separately providedimage pickup part, transmission of information to the video signalprocessing apparatus is not limited to that performed through arecording medium; it can, of course, be performed through wired orwireless communication lines.

Further, in the above description, it is assumed that the processing isperformed by hardware; however, the way the processing is performed isnot limited to this configuration. For example, a configuration may beused in which a signal transferred from the image pickup device 101 isoutput as unprocessed raw data together with header information, such asa configuration selected during capturing, the gain, and the amount of achange in luminance level, transferred from the control section 107; andthe signal is processed in a computer by a video signal processingprogram serving as separate software.

According to this embodiment, when it is judged that residual imagecomponents are included in the image signal of the pixel of interestthat has been subjected to the three-dimensional noise reductionprocess, the factor calculation section 802 specifies large weights forthe two-dimensional noise reduction process, thereby making it possibleto obtain a high-definition image in which residual image components aresuppressed and the occurrence of a discontinuity caused by the noisereduction process is prevented.

1. An imaging system that applies noise reduction processing to imagesignals captured in time series via an image pickup block, comprising: afirst extraction section that extracts a local region that includes apixel of interest from an image signal to be processed; a secondextraction section that extracts, from another image signal captured ata different time from the image signal to be processed, a local regionlocated at almost the same position as the local region extracted by thefirst extraction section; a first noise reduction section that applies afirst noise reduction process to the local region extracted by the firstextraction section, by using the local region extracted by the secondextraction section; a noise estimation section that estimates an amountof noise included in the pixel of interest based on the local regionthat has been subjected to the first noise reduction process in thefirst noise reduction section; a residual image detection section thatdetects a residual image component included in the local region that hasbeen subjected to the first noise reduction process and that is outputfrom the first noise reduction section, based on the amount of noiseestimated by the noise estimation section; and a second noise reductionsection that performs a second noise reduction process on the pixel ofinterest based on the residual image component detected by the residualimage detection section.
 2. An imaging system according to claim 1,wherein the residual image detection section comprises: a calculationsection that calculates the absolute value of the difference between thepixel of interest and the local region that has been subjected to thefirst noise reduction process in the first noise reduction section; anda judgment section that compares the absolute value of the differencecalculated by the calculation section with the amount of noise estimatedby the noise estimation section, to judge whether a residual imagecomponent is included in the pixel of interest.
 3. An imaging systemaccording to claim 2, wherein the residual image detection sectionfurther comprises an adjustment section that adjusts the amount of noisebased on a judgment result for a neighborhood of the pixel of interest.4. An imaging system according to claim 3, wherein the adjustmentsection adjusts the estimated amount of noise to make it smaller when aresidual image component is included in the neighborhood of the pixel ofinterest.
 5. An imaging system according to claim 2, wherein the secondnoise reduction section comprises: a substitution section thatsubstitutes the values of the local region output from the first noisereduction section into the values of the pixel of interest of the imagesignal to be processed, based on a detection result output from theresidual image detection section; and a two-dimensional noise reductionsection that applies the second noise reduction process to the localregion that includes the pixel of interest in the image signal,substituted by the substitution section, based on a judgment resultoutput from the judgment section.
 6. An imaging system according toclaim 1, wherein the second noise reduction section comprises: atwo-dimensional noise reduction section that applies the second noisereduction process to the local region, which includes the pixel ofinterest, extracted by the first extraction section; a factorcalculation section that calculates weighting factors for the imagesignal to be processed and for the local region that has been subjectedto the first noise reduction process in the first noise reductionsection, based on the residual image component detected by the residualimage detection section; and a combining section that multiplies thelocal region that has been subjected to the second noise reductionprocess in the two-dimensional noise reduction section and the localregion that has been subjected to the first noise reduction process inthe first noise reduction section, by the factors calculated by thefactor calculation section, and combines them.
 7. An image processingmethod of applying noise reduction processing to image signals capturedin time series by an image pickup block, comprising: a first step ofextracting a local region that includes a pixel of interest from animage signal to be processed; a second step of storing a predeterminednumber of image signals; a third step of extracting, from another imagesignal captured at a different time from the image signal to beprocessed, a local region located at almost the same position as thelocal region extracted in the first step; a fourth step of applying afirst noise reduction process to the local region extracted in the firststep, by using the local region extracted in the third step; a fifthstep of estimating an amount of noise included in the pixel of interestbased on the local region that has been subjected to the first noisereduction process in the fourth step; a sixth step of detecting aresidual image component included in the local region that has beensubjected to the first noise reduction process in the fourth step, basedon the amount of noise estimated in the fifth step; and a seventh stepof performing a second noise reduction process on the pixel of interestbased on the residual image component detected in the sixth step.
 8. Animage processing program for causing a computer to apply noise reductionprocessing to image signals captured in time series by an image pickupblock, the image processing program causing the computer to execute: afirst process of extracting a local region that includes a pixel ofinterest from an image signal to be processed; a second process ofstoring a predetermined number of image signals; a third process ofextracting, from another image signal captured at a different time fromthe image signal to be processed, a local region located at almost thesame position as the local region extracted in the first process; afourth process of applying a first noise reduction process to the localregion extracted in the first process, by using the local regionextracted in the third process; a fifth process of estimating an amountof noise included in the pixel of interest based on the local regionthat has been subjected to the first noise reduction process in thefourth process; a sixth process of detecting a residual image componentincluded in the local region that has been subjected to the first noisereduction process in the fourth process, based on the amount of noiseestimated in the fifth process; and a seventh process of performing asecond noise reduction process on the pixel of interest based on theresidual image component detected in the sixth process.